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Prologue 


I  am  currently  Involved  in  a  research  effort  to  apply  theoretical 
tools  to  the  rapidly  emerging  area  of  distributed  computing.  I  believe 
that  the  paradigms  of  theory  can  be  applied  to  the  problems  of  concur¬ 
rency  and  distribution  to  yield  a  rich  mathematical  theory  which  will 
provide  a  solid  framework  for  practitioner  and  theoretician  alike  to  use 
in  gaining  greater  understanding  of  this  exciting  and  Important  new  com¬ 
ponent  of  computer  science.  The  long-term  goal  is  to  develop  a  theory  of 
distributed  and  concurrent  computation  analogous  to  the  theory  of  sequen¬ 
tial  computation  that  has  emerged  over  the  past  fifty  years.  Such  an 
effort  will  require  the  involvement  of  a  large  segment  of  the  theoretical 
research  community.  My  goal  here  is  to  present  a  few  examples  of  work  to 
date  which  give  the  flavor  of  my  work  and  to  point  out  areas  in  which 
interaction  with  practitioners  could  be  particularly  beneficial. 
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1.  The  Elements  of  a  Theory 

By  a  theory,  I  mean  the  result  of  removing  ambiguity  and  uncertainty 
in  the  statement  of  a  problem  so  that  precise,  rigorous  statements  about 
it  can  be  made  and  verified.  Abstraction,  the  process  of  eliminating  un¬ 
interesting  detail,  is  an  essential  part  of  the  process  of  constructing  a 
theory,  for  it  permits  the  problem  to  be  reduced  to  a  tractable  and  compre¬ 
hensible  size  from  which  significant  new  insights  can  be  obtained.  The 
danger  in  studying  abstractions  is  that  certain  essential  elements  may 
have  been  discarded  along  with  the  inessential,  resulting  in  conclusions 
that  are  valid  for  the  abstraction  but  nevertheless  may  not  hold  for  the 
original,  motivating,  real-world  problem.  The  only  answer  to  this  diffi¬ 
culty  that  I  know  of  is  to  keep  always  aware  of  the  problem,  constantly 
question  the  assumptions  being  made,  check  the  conclusions  against  reality, 
and  study  a  variety  of  different  abstractions. 

The  words  "distributed  computing"  suggest  many  different  new  aspects 
of  computer  science  introduced  by  the  advent  of  networks  of  independent 
communicating  processors,  and  different  kinds  of  theories  are  likely  to 
emerge  from  each.  A  few  that  we  have  investigated  to  date  include  problems 
of  concurrency  and  synchronisation,  mechanisms  and  protocols  for  interpro¬ 
cessor  communication,  tools  for  performance  measurement  and  analysis,  effects 
of  geometry  and  topology,  and  reliability  considerations.  By  way  of  contrast, 
other  important  aspects  which  I  have  not  addressed  include  programming  issues 
such  as  naming  and  binding,  nor  have  I  looked  at  protection  and  security 


issues . 
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I  give  below  some  brief  sketches  of  my  work  on  a  variety  of  these 
topics  to  illustrate  the  kinds  of  questions  I  have  been  asking  and  the 
sorts  of  results  that  can  be  obtained. 

2 .  Synchronization  Problems 

My  first  work  in  this  area  concerned  questions  of  synchronization  of 
concurrent  processes  in  environments  in  which  keeping  small  the  amount  of 
information  exhcanged  by  processes  was  an  important  consideration.  In 
|PF],  Gary  Peterson  and  I  looked  into  the  amount  of  "communication  space" 
that  was  required  to  solve  the  classical  "critical  section"  problem  for  n 
processes  which  communicated  using  the  "message  board"  model  in  which  each 
process  has  a  communication  variable  which  only  it  can  write  and  every  other 
process  can  read.  Even  without  reference  to  clocks  or  other  timing  considera¬ 
tions,  we  were  able  to  exhibit  a  fair  solution  which  used  only  a  constant 
amount  of  comnunication  space  per  process,  independent  of  the  total  number 
of  processes  in  the  system. 

In  f BFJLP  I,  we  looked  at  the  same  problems  in  a  much  stronger  (and  less 
"distributed")  model  in  which  processes  communicated  via  shared  memory  ac¬ 
cessed  through  a  general  "test-and-set"  Instruction.  Even  with  such  a  power¬ 
ful  model,  we  were  able  to  prove  non-trivial  lower  bounds  on  the  rate  at  which 
communication  space  must  grow  as  the  number  of  processes  to  be  synchronized 
increases,  when  various  fairness  constraints  are  Imposed.  In  particular,  n/2 
values  are  required  for  any  starvation-free  solution  for  n  processes,  and 
n+1  are  needed  if  In  addition  the  solution  has  bounded-waiting.  To  show  that 
these  bounds  are  essentially  best-possible,  we  constructed  actual  algorithms 


that  nearly  achieve  them.  While  the  algorithms  themselves  may  be  of 
limited  practical  interest,  they  involve  several  novel  programming 
"tricks"  and  internal  protocols  which  may  be  useful  in  other  contexts. 

3.  Models  of  Behavior 

Virtually  all  of  the  work  on  the  critical-section  problem,  our  own 
included,  suffered  from  the  fact  that  the  problem  was  stated  in  terms  of 
particular  desired  internal  properties  of  the  computing  system  (e.g.  mutual 
exclusion)  rather  than  on  observable  behavior.  It's  nice  to  be  able  to 
quantify  the  cost  of  achieving  mutual  exclusion,  but  it's  even  nicer  to 
be  able  to  state  a  problem  for  which  mutual  exclusion  is  necessary.  Looked 
at  from  another  point  of  view,  one  does  not  want  the  statement  of  a  computing 
problem  to  unduly  constrain  the  system  designer.  For  example,  the  specifi¬ 
cations  for  an  airline  reservation  system  might  state  the  numbers  and  loca¬ 
tions  of  terminals,  functional  requirements,  expected  response  times,  etc., 
but  it  should  not  prescribe  things  like  the  number  and  locations  of  internal 
nodes  in  the  network  or  its  topology. 

These  considerations,  together  with  the  need  for  a  formal  model  appro¬ 
priate  for  performing  time  analyses,  led  Nancy  Lynch  and  me  to  define  an 
abstract  notion  of  the  behavior  of  a  distributed  system  based  only  on  ob¬ 
servable  input  and  outputs  to  the  system  and  not  on  the  internal  workings 
of  the  system  [LFal. 

4 .  Timing  Analysis 


As  in  the  case  of  sequential  complexity  theory,  it  makes  sense  to 
measure  both  "worst-case"  running  times  and  "expected"  times  under  some 


assumed  probability  distribution.  The  advantage  of  the  former  is  that  it 
is  independent  of  dif f icult-to-defend  probability  assumptions,  gives  ab¬ 
solute  bounds  concerning  performance  (and  hence  correctness)  in  a  real-time 
environment,  and  is  often  far  more  tractable  mathematical ly .  The  latter  of 
course  may  reflect  the  observable  realities  far  better. 

In  the  case  of  distributed  algorithms,  however,  it  is  not  always  so 
clear  just  what  to  measure,  for  non-terminating  programs  are  the  rule  and 
there  may  be  a  trade-off  between  performance  measures  such  as  throughput 
and  response  time.  We  suggest  some  examples  of  appropriate  timing  measures 
for  the  particularly  simple  arbiter  problem  in  T FLa  ] ,  and  we  expand  on  those 
ideas  in  the  revised  version  of  that  paper  fLFb  I.  These  are  worst-case  kinds 
of  analyses.  Nancy  Lynch,  Ed  Lazowska,  Pat  Jacobson  and  I  are  currently  try¬ 
ing  to  perform  an  expected-time  analysis  of  these  same  systems  for  comparison 
using  queuing  theory  techniques. 


5.  Reliability  and  Fault-Tolerance 

In  [FLBBl,  we  looked  at  the  seemingly  minor  generalization  of  the  criti 
cal  section  problem  to  permit  a  maximum  of  some  number  f  of  processes  to 
be  simultaneously  in  their  critical  sections  but  not  more.  Obvious  generali¬ 
zations  of  1-criticnl  section  solutions  become  unattractive  when  we  require 
in  addition  that  the  system  "tolerate"  the  failure  of  a  limited  number  of 
processes.  The  kind  of  fault  we  consider  is  a  process  simply  ceasing  to  take 
further  steps,  but  it  does  not  announce  this  fact  to  the  other  processes  in 
any  way.  Since  we  do  not  include  clocks  in  our  model,  "time-outs"  are  im¬ 


possible,  so  there  is  no  way  for  one  process  to  tell  whether  another  has 
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failed  or  is  just  running  very  slowly.  Our  main  results  are  that  there  is 
an  algorithm  for  solving  this  problem  which  tolerates  limited  process  fail¬ 
ure,  satisfies  a  suitably-generalized  notion  of  "bounded-waiting",  and  uses 
only  0(n)  values  of  shared  conmunicat ion  memory.  For  FIFO  fairness  prop- 
erty,  we  need  only  0(n  (log  n)  )  values  of  shared  memory.  Again  we  hope 
the  principles  used  in  the  design  of  these  algorithms,  if  not  the  algorithms 
themselves,  will  be  exportable  to  other  problems  of  reliable  concurrent  com¬ 
putation. 

Michael  Rabin,  Udi  Manber  and  I  in  some  current  research  are  investi¬ 
gating  another  problem  in  which  the  key  technical  difficulty  is  to  achieve 
reliable  operation  of  the  non-failing  portion  of  the  system.  Here  our  model 
is  a  large  number  of  independent  processes  all  simultaneously  searching  a 
directed  graph  represented  in  list-structured  memory.  Because  of  memory 
constraints,  the  problem  is  not  for  processes  to  cooperate  with  each  other 
but  simply  for  them  to  stay  out  of  each  other's  way.  The  results  of  our 
efforts  so  far  are  a  surprisingly  non-trivial  algorithm  for  solving  this 
problem  in  small  space  and  a  new  kind  of  "locking"  protocol. 

A  third  piece  of  current  work,  this  time  joint  with  Nancy  Lynch  and 
Leslie  Lamport,  concerns  the  paper  of  Pease,  Shostak,  and  Lamport  I PSI.  I  on 
coping  with  "malicious  failure"  in  which  the  failed  processes  might  continue 
to  take  steps  but  produce  erroneous  and  unpredictable  results.  The  algorithm 
of  f PSLj  for  achieving  agreement  in  the  presence  of  such  faults  works  in 
stages  or  "rounds"  in  which  each  process  exchanges  Information  with  each 
other.  To  protect  against  a  maximum  of  k  faulty  processes,  the  algorithm 
requires  k+1  rounds  (and  hence  time  at  least  k+1  ).  Although  a  faster 


algorithm  would  be  very  desirable,  we  can  show  that  no  algorithm  can  solve 
this  problem  in  fewer  than  k+1  rounds.  We  think  similar  results  can  also 
be  obtained  for  the  cases  where  messages  are  non-forgable  and  where  the  pro¬ 
cesses  are  asynchronous. 

6.  Resource-Placement  Problems 

The  allocation  of  resources  to  processes  in  a  distributed  system  is 
particularly  challenging  and  seems  to  underlie  many  other  problems  of  dis¬ 
tributed  computation.  We  imagine  a  fixed  number  of  resources  or  "tickets" 
for  which  requests  may  originate  at  various  points  in  Lbe  system.  The  basic 
property  of  a  ticket  system  is  that  it  never  grant  more  requests  than  there 
are  tickets  available.  Other  possibly  desirable  properties  are  that  a  re¬ 
quest  not  get  denied  when  tickets  are  available  elsewhere  in  the  system, 
and  that  the  system  be  able  to  accept  the  return  of  tickets. 

In  joint  work  with  Nancy  Lynch  and  Nancy  Oriffeth,  we  began  by  investi¬ 
gating  a  number  of  ticket  algorithms  for  networks  of  various  topologies  but 
had  difficulty  in  comparing  them.  This  led  us  to  abstract  further  to  a  very 
simple  problem:  assume  the  network  is  a  complete  binary  tree,  that  requests 
originate  at  randomly  chosen  leaves,  and  that  requests  are  matched  to  tickets 
in  an  optimal  way.  The  problem  Is  then  to  find  what  initial  placement  of 
tickets  on  the  nodes  of  the  tree  leads  to  the  least  expected  distance  between 
a  request  and  the  ticket  which  satisfies  that  request.  Rather  surprisingly, 
we  found  that  the  expected  distance  for  the  optimal  placement  is  constant  as 
long  as  the  number  of  tickets  (and  requests)  is  at.  least  proportional  to  the 
number  of  leaves.  This  suggests  that,  at  least  in  certain  circumstances. 


algorithms  which  dynamically  move  the  tickets  around  the  system  can  at  best 
achieve  only  limited  improvements  over  that  obtainable  by  static  placement 
algorithms . 

7.  Conclusion 

The  work  described  above  spans  a  considerable  number  of  topics  of  prac¬ 
tical  importance  in  distributed  computing.  I  feel  it  essential  to  learn 
more  about  the  practical  issues  in  order  to  form  more  meaningful  and  relevant 
abstractions.  At  the  same  time,  I  feel  it  valuable  to  share  the  insights  I 
have  already  attained  through  these  studies  with  others  in  the  field.  This 
workshop  seems  an  ideal  forum  for  such  interchange. 
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